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I'm not a human: Breaking the Google reCAPTCHA 
https://www.blackhat.com/docs/asia-16/materials/asia-16- 


Sivakorn-Im-Not-a-Human 


Al ШИШ 


Deep Learning-based 
Handwriting Generation Machine 
— LiChuang@CFCA 
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精度 可 达 0.02mm ,以 3D 打 印 为 技术 基础 
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https://www.pyimagesearch.com/2016/06/20/detecting-cats-in-images-with-opencv/ 
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https://www.pyimagesearch.com/2016/06/20/detecting-cats-in-images-with-opencv/ https://www.wired.com/story/tried-to-beat-face-id-and-failed-so-far/ 
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build/examples/cpp_classification/classification.bin | 
models/bvlc reference caffenet/deploy.prototxt | 

models/bvlc reference caffenet/bvlc reference caffenet.caffemodel V 
data/ilsvrc12/imagenet mean.binaryproto V 
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21 m examples/images/cat.jpg 


https://github.com/BVLC/caffe/tree/master/examples/cpp classification 
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/build/examples/cpp classification/classification.bin V 
models/bvlc reference caffenet/deploy.prototxt | 

models/bvlc reference caffenet/bvlc reference caffenet.caffemodel V 
data/ilsvrc12/imagenet mean.binaryproto V 

data/ilsvrc12/synset words.txt | 

СО poexamples/images/cat.jpg 


Output: 


---------- Prediction for examples/images/cat.jpg ---------- 
0.3134 - "n02123045 tabby, tabby cat" 
0.2380 - "n02123159 tiger cat" 

0.1235 - "102124075 Egyptian cat" 

0.1003 - "n02119022 red fox, Vulpes vulpes" 

0.0715 - "n02127052 Iynx, catamount" 


https://github.com/BVLC/caffe/tree/master/examples/cpp classification 


# bullog [Original Image] 

./classification.bin models/bvlc reference caffenet/deploy.prototxt 
models/bvlc reference caffenet/bvlc reference caffenet.caffemodel 
data/ilsvrc12/imagenet mean.binaryproto 

data/ilsvrc12/synset words.txt 

./poc_samples/bulldog 

---------- Prediction for ./рос samples/bulldog ————— 

0.5111 - "n02108915 French bulldog" 


# Threat Example 2 -- Evasion attack 
# bulldog sh | cause classification to misclassify to an arbitr: 
# [Here the classification produced a class "Flying Pig" made 
./classification.bin models/bvlc reference caffenet/deploy.pro 
models/bvlc reference caffenet/bvlc reference caffenet.caffe 
data/ilsvrc12/imagenet mean.binaryproto 
data/ilsvrc12/synset words.txt 
./poc_samples/bulldog fp 
---------- Prediction for ./poc samples/bulldog fp ---------- 

0.98 - "n03770679 flyingpig” 


# Threat Example 1 -- DoS attack 

# bulldog crash [ cause classification binary to crash] 
./classification.bin models/bvlc reference caffenet/deploy.prototxt 
models/bvlc reference caffenet/bvlc reference caffenet.caffemodel 
data/ilsvrc12/imagenet mean.binaryproto 

data/ilsvrc12/synset words.txt 

./poc_samples/bulldog crash 

---------- Prediction for ./poc samples/bulldog crash ---------- 
Segmentation fault (core dumped) 


# Threat Example 3 -- Exploitation Attack 


# bulldog sh [ cause classification to generate a local shell] 

./classification.bin models/bvlc reference caffenet/deploy.pr: 

models/bvlc reference caffenet/bvlc reference caffenet.caff« 

data/ilsvrc12/imagenet mean.binaryproto 

data/ilsvrc12/synset words.txt 

.Грос samples/bulldog sh 

---------- Prediction for ./poc samples/bulldog sh ---------- 

$ uname -a 

Linux ctf-box 4.4.0-31-generic #50~14.04.1-Ubuntu SMP 

Wed Jul 13 01:07:32 UTC 2016 x86 64 x86 64 x86 64 GNU/L 
xi 


# bullog [Original Image] 

./classification.bin models/bvlc reference caffenet/deploy.prototxt 
models/bvlc reference caffenet/bvlc reference caffenet.caffemodel 
data/ilsvrc12/imagenet mean.binaryproto 

data/ilsvrc12/synset words.txt 


# Threat Example 1 -- DoS attack 

# bulldog crash [ cause classification binary to crash] 
./classification.bin models/bvlc reference caffenet/deploy.prototxt 
models/bvlc reference caffenet/bvlc reference caffenet.caffemodel 
data/ilsvrc12/imagenet mean.binaryproto 

data/ilsvrc12/synset words.txt 

./poc_samples/bulldog crash 

---------- Prediction for ./poc samples/bulldog crash ---------- 
Segmentation fault (core dumped) 


# Threat Example 2 -- Evasion attack 

# bulldog sh | cause classification to misclassify to an arbitr: 
# [Here the classification produced a class "Flying Pig" made 
./classification.bin models/bvlc reference caffenet/deploy.pro 
models/bvlc reference caffenet/bvlc reference caffenet.caffe 
data/ilsvrc12/imagenet mean.binaryproto 

data/ilsvrc12/synset words.txt 

./poc_samples/bulldog fp 

---------- Prediction for ./poc samples/bulldog fp ---------- 

0.98 - "n03770679 flyingpig” 


# Threat Example 3 -- Exploitation Attack 


# bulldog sh [ cause classification to generate a local shell] 

./classification.bin models/bvlc reference caffenet/deploy.pr: 

models/bvlc reference caffenet/bvlc reference caffenet.caff« 

data/ilsvrc12/imagenet mean.binaryproto 

data/ilsvrc12/synset words.txt 

.Грос samples/bulldog sh 

---------- Prediction for ./poc samples/bulldog sh ---------- 

$ uname -a 

Linux ctf-box 4.4.0-31-generic #50~14.04.1-Ubuntu SMP 

Wed Jul 13 01:07:32 UTC 2016 x86 64 x86 64 x86 64 GNU/L 
xi 


# bullog [Original Image] 

./classification.bin models/bvlc reference caffenet/deploy.prototxt 
models/bvlc_reference_caffenet/bvlc reference caffenet.caffemodel 
data/ilsvrc12/imagenet mean.binaryproto 

data/ilsvrc12/synset words.txt 
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SEGMENTA ION FAULT! 
# Threat Example 1 | p 

# bulldog crash | са 
./classification.bin 


Мо crash] 
#Caffenet/deploy.prototxt 


models/bvlc referen 2 caffenet.caffemodel 
data/ilsvrc12/imageHeteme IA DLC to 

data/ilsvrc12/synset 

./poc_samples/bulldc 

---------- Prediction f@ rash ---------- 


Segmentation fault 


# Threat Example 2 -- Evasion attack 

# bulldog sh | cause classification to misclassify to an arbitr: 
# [Here the classification produced a class "Flying Pig" made 
./classification.bin models/bvlc reference caffenet/deploy.pro 
models/bvlc reference caffenet/bvlc reference caffenet.catffe 
data/ilsvrc12/imagenet mean.binaryproto 

data/ilsvrc12/synset words.txt 

./poc_samples/bulldog fp 

---------- Prediction for ./poc samples/bulldog fp ---------- 

0.98 - "n03770679 flyingpig” 


# Threat Example 3 -- Exploitation Attack 


# bulldog_sh [ cause classification to generate a local shell] 

./classification.bin models/bvlc reference caffenet/deploy.pr: 

models/bvlc reference caffenet/bvlc reference caffenet.caff« 

data/ilsvrc12/imagenet mean.binaryproto 

data/ilsvrc12/synset words.txt 

.Грос samples/bulldog sh 

---------- Prediction for ./poc samples/bulldog sh ---------- 

$ uname -a 

Linux ctf-box 4.4.0-31-generic #50~14.04.1-Ubuntu SMP 

Wed Jul 13 01:07:32 UTC 2016 x86 64 x86 64 x86 64 GNU/Li 
xi 


# bullog [Original Image] 

./classification.bin models/bvlc reference caffenet/deploy.prototxt 
models/bvlc_reference_caffenet/bvlc reference caffenet.caffemodel 
data/ilsvrc12/imagenet mean.binaryproto 

data/ilsvrc12/synset words.txt 
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SEGMENTA ION FAULT! 
# Threat Example 1 | p 

# bulldog crash | са 
./classification.bin 


Мо crash] 
#Caffenet/deploy.prototxt 


models/bvlc referen 2 caffenet.caffemodel 
data/ilsvrc12/imageHeteme IA DLC to 

data/ilsvrc12/synset 

./poc_samples/bulldc 

---------- Prediction f@ rash ---------- 


Segmentation fault 


# Threat Example 2 -- Evasion a 
# bulldog_sh | cause classificati 
# [Here the classification produ 
./classification.bin models/bvlc rg 
models/bvlc reference caffenet/ 
data/ilsvrc12/imagenet mean. biı 
data/ilsvrc12/synset words.txt 
./poc_samples/bulldog fp 
---------- Prediction for ./poc samples/bulldog Гр ---------- 
0.98 - "n03770679 flyingpig” 


# Threat Example 3 -- Exploitation Attack 


# bulldog_sh [ cause classification to generate a local shell] 

./classification.bin models/bvlc reference caffenet/deploy.pr: 

models/bvlc reference caffenet/bvlc reference caffenet.caff« 

data/ilsvrc12/imagenet mean.binaryproto 

data/ilsvrc12/synset words.txt 

.Грос samples/bulldog sh 

---------- Prediction for ./poc samples/bulldog sh ---------- 

$ uname -a 

Linux ctf-box 4.4.0-31-generic #50~14.04.1-Ubuntu SMP 

Wed Jul 13 01:07:32 UTC 2016 x86 64 x86 64 x86 64 GNU/Li 
xi 


./classification.bin models/bvlc reference caffenet/deploy.prototxt 
models/bvlc reference caffenet/bvlc reference caffenet.caffemodel 
data/ilsvrc12/imagenet mean.binaryproto 

data/ilsvrc12/synset words.txt 

./poc_samples/bulldac 


0.5111 - "n0210 
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./classification.bin ffenet/deploy.prototxt 


models/bvlc referen _caffenet.caffemodel 
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data/ilsvrc12/synset 
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---------- Prediction fom rash ---------- 


/classification.bin models/bvlc | 

models/bvlc reference caffenet/ 

data/ilsvrc12/imagenet mean. biı 

data/ilsvrc12/synset words.txt 

.Грос samples/bulldog fp 

---------- Prediction for ./poc samples/bulldog fp ---------- 
0.98 - "n03770679 
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https://www.cs.toronto.edu/-ranzato/publications/taigman cvpr14.pdf 
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MNIST Deep Learning Model Input Requirement 
(pixel x pixel) 
MNIST 28х28 
http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf ImageNe t 
ImageNet AlexNet 227x227 
GoogleNet, УСС 224x224 
I ResNet 224x224 
https://github.com/BVLC/caffe/tree/master/examples/cpp classification 
NVIDIA PX DAVE-2 NVIDIA 
መመ ڪڪ‎ | 1 =| | | DAVE-2Self-Driving 200x66 


ge ` 


https://images.nvidia.com/content/tegra/automotive/images/2016/solutions/pdf/end-to-end-dl-using-px.pdf 
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NVIDIA Sample Self-Driving 
Models 


NVIDIAPX2 
DAVE -2Selt- 200x060 
Driving 


T 


Caffe? 


https://images.nvidia.com/content/tegra/automotive/images/2016/solutions/pdf/end-to-end-dl-using-px.pdf 


NVIDIA Recommended Ecosystem 
Camera Vendors 


Filr 
A310 320x240 
A615 640x480 
Leopard 
LI-ARO231 1920x1208 


578526100 1920х1208 


https://developer.nvidia.com/drive/ecosystem 
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def read tensor from image file(file name, input height=299, input width=299, 


| 
2 input mean=0, input std=255): 
5 | input name = "file reader" 
41 output name = "normalized" 
5 file reader = tf.read file(file name, input name) 
6 | if file name.endswith(".png"): 
7% image reader = tf.image.decode png(file reader, channels = 3, 
8 name='png reader’) 
TensorFlow 9 | elif file name.endswith(".gif"): 
Example 10 j image reader = tf.squeeze(tf.image.decode gif(file reader, 
11 name='gif reader')) 
12 | elif file name.endswith(".bmp"): 
13 ЇЇ Image reader = tfimage.decode bmp(file reader, name='bmp reader?) 
14 else: 
M image reader = tf.image.decode jpeg(file reader, channels = 3, 
16 ፪ name='jpeg_reader') 
НТ float caster = tf.cast(image reader, tf.float32) 
19 resized = tf.image.resize bilinear(dims expander, [input height, input width]) 
20 normalized = tf.divide(tf.subtract(resized, [input mean|]), [input std]) 
" sess = tf.Session() 
221 result = sess.run(normalized) 
23 1 


24 return result 
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Ke, 
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DeepDetect Example 


int read file(const std::string &fname) 
{ 


cv::Mat img = cv::imread(fname, bw ? CV LOAD IMAGE GRAYSCALE 
CV LOAD IMAGE COLOR); 
if (img.empty()) 


LOG(ERROR) << "empty image"; 
return -1; 


_imgs_size.push_back(std::pair<int,int>(img.rows,img.cols)); 
cv::Size size( width, height); 


cv::resize(img,rimg,size,0,0,CV_INTER CUBIC); 
_imgs.push_back(rimg); 
return 0; 
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Scaling is supposed to preserve the visual features of an image 


and thus does not change its semantic meaning. 
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Value at (x,y) = 
Sum of the value at each spot multiplied 
by the area of the rectangle divided by the 
totalarea of all four rectangles 


https://en.wikipedia.org/wiki/Bilinear interpolation 


f(i+u,j+v) = (1-u)(1-v)f(i,]) + (1-u)vf(i,j-- 1) + u(1-v)f(1+1,)) + uvf(i+l,j+1) 
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An image is a matrix, 
and scaling is done 


with convert matrix ШЇЇ (|| | 


Attack Image Target Image 
(matrix A) (matrix T) 


Ни I! 


An image is a matrix, L LL » 
and scaling is done са , => ны 7 3 
with convert matrix MI | 
Attack Image Target Image 
(matrix A) (matrix T) 


N [p,m] х A[m,n] x C[n,q] = Tlp,q] 
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Input Image Attack Image Target Image 
(matrix S) (matrix A) (matrix T) 


S[]+ A П-АП АПхСП- ТП 


Ни I! 


ол - - ይላ ЫС 


Input Image Attack Image Target Image 
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